Posted 7 August 2011
The Scientific Alliance, UK
5 August 2011
Peer review and scepticism
Peer review, as most of you will be well aware, is the process by which scientific papers are screened before publication. In what sounds an eminently sensible system, journal editors pass on submitted papers to experts in the appropriate field and decide, based on the comments they get back, whether to accept or reject them or ask for revisions. The idea is that substandard or faulty work will be filtered out and that only research which has been properly conducted and written up will enter the literature. Journals maintain their reputation for publishing ‘proper’ research and, since researchers’ careers depend on publication, they toe the line and only write papers which are of good enough quality.
This is referred to by some as the ‘gold standard’; a paper published in a peer-reviewed journal should be a reliable and unquestioned addition to the body of knowledge. In the majority of cases, where small, incremental advances are made, all is probably well. But there is an inbuilt conservatism to the entire system which can discourage work which questions received wisdom. After all, the reviewers will almost certainly subscribe to the mainstream view on particular issues and, indeed, may well have helped to develop the evidence for them. To compound this, peer review is conducted anonymously.
While this will weed out some bad papers, it is no guarantee that those published will report good scientific work. It is not unknown for papers to be withdrawn from even the most prestigious journals, following valid public criticism from other scientists. There are even occasional cases of deliberate fraud: science is no more immune to this than any other walk of life. It is arguable that the best form of peer review is actually post publication, with the reviewers being self-selected and criticisms being made and answered in public.
In the meantime, the inbuilt conservatism of the process – combined with a grant-making system which tends to encourage cutting-edge research only in certain directions – acts as a brake on some truly original but controversial lines of study. At the same time, research with aims consistent with mainstream views is very likely to be published. Scepticism, the very bedrock of science, is encouraged in reviewers and discouraged in researchers.
Lack of balance in the direction of research can also create or reinforce concers. In our sophisticated and wealthy modern societies, precaution is the order of the day. As longevity increases, our attitude to risk has changed, making us hyper-sensitive to reports of hazards which might be lying in wait for the unwary. Hence the uproar over GM crops in the mid and late ‘90s when a suggestible public found it quite plausible that scientists tinkering with something as basic as food might have created crops with significant safety issues. This was unfortunately compounded by the work having being done by another of our modern bogeymen, the faceless multinational corporation. Even worse, underlying this was – horror of horrors – the profit motive (the public sector, as is well known, is driven only by the purest of motives and therefore can do no harm).
Equally with the MMR vaccine: a dubious piece of work was seized upon by the media and created a scare about the safety of vaccinations. In the case of the GM crops story, European farmers have lost out and further damage has been done to EU competitiveness, but the MMR scare temporarily increased the incidence of potentially harmful childhood diseases. This is ironic, since the parents who did not have their babies vaccinated mistakenly thought they were protecting them.
In some cases, commonsense prevails. Despite repeated stories about the possible dangers of mobile phone radiation, their use has continued to mushroom. The risk side of the equation is, in most people’s minds, heavily outweighed by the benefits. Even evidence put forward by some researchers that carrying mobile phones in pockets reduces sperm quality has not spooked the great majority of men.
None of this, of course, has much to do with anything published in peer-reviewed journals. Scares were created, by and large, by activists, some of whom were scientists. Occasionally, something very controversial is published in the mainstream literature. For example, the infamous incomplete Pusztai study purporting to show safety problems with mice fed GM potatoes was eventually published in the Lancet, but the publication itself was heavily criticised by many scientists (see GM controversy intensifies).
But some topics are subject to almost constant, albeit low-level, concern. Pesticides is one. A recent on-line (but peer-reviewed) article was entitled Widely Used Pesticides with Previously Unknown Endocrine Activity Revealed as in Vitro Antiandrogens. The message is that these pesticides may play a significant role in the ‘widespread decline in male reproductive health’. The results of the experiments are undoubtedly true, but incubating cells with particular chemicals is a far cry from ingesting minute traces on food. The dose makes the poison and, in the often quoted words of Bruce Ames, There are more rodent carcinogens in a single cup of coffee than potentially carcinogenic pesticide residues in the average American diet in a year, and there are still a thousand chemicals left to test in roasted coffee (The causes and prevention of cancer: the role of the environment).
Climate change is another topic where it seems that all news is bad news. Starting from the premise that climate change is man-made and bad, researchers naturally do studies to define just how bad. And, because the only thing you can do objectively in climate science is to record temperatures, rainfall and so on, experiments are replaced by computer modelling, the results of which have become accepted as a true picture of what will happen over the next century. Papers written on the basis of these studies are peer-reviewed, and the vast majority add further bricks to the edifice of dangerous climate change.
All scientific papers deserve to be looked at critically, or even sceptically. Human intelligence remains the best defence against bad science as well as the means to advance knowledge. Science progresses via (usually constructive) criticism and no researcher should object to that. Peer review, by tending to reinforce particular points of view, increases the need for open-minded reading; it doesn’t mean we can accept published papers as the unquestionable truth.